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APPLTCATTON FOR LRTTKRS PA , 



Field Of The Tnvenrinn 

Ihis inventim relates to surgical systems in general, and more particularly 
to surgical targeting systems. 

t 

Backp-OUfid Of The Tnvpntinn 

Many medical procedures must be canied out at an interior site which is 
normally hidden from the view of tiie physician. In these situations, the 
pltysician typically uses some sort of scanning device to examine the patient's 
anatomy at the inteiior site prior to, and in preparation for, conducting the 
actual procedure itself. These scamiing devices typically include MRI devices, 
CT scanners. X-ray machines, ultrasound devices and the Uke, and serve to 
provide the physician with a preliminaiy knowledge of the patient's internal 
anatomical structure prior to commencing the procedure. The physician can 
Aen use diis xufoxmation to plan the procedure in advance, taking into account 
patient-specific anatomical structure. In addition, the physician can also use 
tiie information obtained from these scans to more precisely identify the 
location of selected structures (e.g. tumors and the like) which may tiiemselves 
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be located iviliiin internal organs or other intemd to As a 

result, tiie physician can more easily "zero in** on such selected structures 
during the subsequent procedure, with less trauma to host organs or other 
internal body structures. FurAemior^ in many cases the structures of interest 
may be qmte small and di£5cult to identify vvidi the naked eye. In these cases, 
preliminary scanning of the patient's internal anatomy using high resolution 
scanning devices can help the physician locate such structures during the 

iLl 

subsequent procedure. 

In general, scanning devices of the sort described above tend to generate 
a series of two-dimensional (i.e., *'2-D") images of the patient's anatomical 
structure. For example, CT scanners generate 2-D images which are viewed 
directly by the physician. By viewing a plurality of ^ese 2-D images, the^ 
physidan can mentally generate a three-dimensional (i.e., "3-D") sense of the 
patient^s anatomical structure. 

Some scanning devices create large numbers of 2-D images during the 
scanning process, with each 2-D image representing a plane or slice taken 
through the scanned stracture. Furthermore, some scanning devices also have 
associated computer hardware and software for building a 3-D computer 
model of the patient! s scanned structure using a plurality of these 2-D images. 
For example, some MRI devices and some CT scanners have such associated 
computer hardware and software. In these cases, an operator using this 
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scanhing equipment and associated computer haidwaie and sofhvaie can 
CTcate a number of 2-D images, assemble them into a 3-D computer model of 
the scanned stracturc; and then generate various images of that 3-D conqiuter 
model as seen from various angles so as to enhance understanding of the 
patient's anatomical structure. 

While the infoxmation generated by the aforementioned scanning devices 
is generally of great benefit to physicians, certain significant lunitations still 
exist. For one thmg, it can be very difBcuit for a physician to physically 
locate a particular anatomical structure during a medical procedure, even when 
that structure is leadify identifiable in a scanned image. This may be because 
Ae structure is vco^ snudl and difficult to see vrith the naked ^^^^ or because 
it is not readily visible against a particular background, or because it is itself 
located ivitfain an internal cngan or other internal body stmctore, etc. For 
another thing, even vdien the structure of interest is successfiilly located by the 
physician, it can sometimes still be extremely difficult for the physician to 
reliably direct a medical instrument to that stracture. This may be because the 
structure is quite small and difficult to target accurately, or because 
int^vening bo^ structure mandates a complex approach, etc. 
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Ol^ectfi Of The Invention 

Accordingly, one object of the present invention is to provide a surgical 
taigeting system to facilitate locating a particular anatomical structure during 
a medical procedure. 

Anodier object of the present invmtion is to provide a video-based surgical 
targeting system to facilitate locating a particular anatomical structure during 
a medical procedure. 

And another object of the present invention is to provide a video-based 
surgical targeting system which permits a series of patient-specific 2-D 
images (obtained by scanning patient anatomy using one or more scantung 
devices of the type described above) to be assemUed into a 3-D computer 
model of the patient's scanned structure. 

Still another object of the present invention is to provide a video-based 
smgical taigeting system which allows a physician to view the aforementioned 
patient-specific 2-D images on a display in any desired access sequence. 

And another object of the present invention is to provide a video-based 
surgical targeting system which allows a physician to assemble a series of 
patimt-specific 2-D images into a patient-specific database, and then to 
generate virtual images fi^om the aforementioned patient-specific database, as 
seen firom any desired virtual camera position, for viewing on a display. 
Yet another object of the present invention is to provide a video-based 
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smgical taigeting system which allows a physician to generate virtual images 
from the afoiementioned 3-D compater model, as seen from any desired 
virtual camera position, for viewing on a display. 

And another object of die present invention is to provide a video-based 
smgical taigeting system which permits a physician to place virtual planning 
maikers about any sites of interest while viewing one or more of die 
aforementioned padent-specific 2*D images, widi those virtual planning 
maiicers then being incorporated into the 3-D computer model, whereby those 
virtual planning markers can be displayed in dieir appropriate 3-D positions 
when generating virtual images of the 3-D computer model. 

Still another object of the present invention is to provide a video-based 
smgical targeting system which permits a physician to place virtual planning 
. markers about any sites of interest while viewing virtual images of the 3-D 
computer model, with those virtual planning markers then being incorporated 
into thci 3-D computer model, whereby those virtual planning markers can be 
displayed in their appropriate positions when generating subsequent virtual 
images of die 3-D computer model. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which permits a physician to place virtual planning 
markers about any sites of interest while viewing virtual images of the 3-D 
computer model, widi those virtual planning markers then being incorporated 
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into tiie 3-D compiiter modd and into tiie database of 2-D images, whereby 
those viitual planning markers can be displayed in their appropriate positions 
when snbsequently generating virtual images of die 3-D computer model or 
vdien subsequent^ diq>laying 2-D images from the patient-specific database. 

Yet another object of the present invention is to provide a video-based 
surgical taigeting system which pemiits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be displi^ed to a physician, and which 
permits a virtual image generated from the 3-D computer model to be 
displayed to a physician, according to the directive of the physician. 

Anotiier object of the present invention is to provide a video-based surgical 
taigeting system which pemiits a real image obtained by a real-time imaging 
device (e.g. a video camera) to be appropriately merged with a corresponding 
virtual image generated from the 3-D computer model. 

And another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be appropriately merged with a 
conresponding virtual image generated from die 3-D computer model, whmby 
tiie two images will be in registration with one another. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
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virtual image generated from the 3-D computer model, whereby the two 
images will be in registration wifli one another, and whereby the physician can 
choose to display either one of the two images exclusive of the other, or a 
composite of both images simultaneously. 

Yet another object of die present invention is to provide a video-based 
smgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby the physician can modify the virtual image generated from die 3-D 
computer model as needed, by clipping or fading, so as to expose the virtual 
planning markers to view, with the virtual planning markers being 
superimposed on the real image generated by the realrtime viewing device. 

And another object of the present invention is to provide a video-based 
smgical targeting system which permits a real image generated by a real-time 
imaging device (e.g. a video camera) to be m^ged with a couesponding 
virtual image graerated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby the physician can modify die virtual image generated from the 3-D 
computer model so as to expose only the virtual planning markers to view, 
with tiie virtual planning markers being superimposed on the real image 
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generated by die real-time imagmg device. 

Still another object of the preset invention is to provide a video-based 
surgical targeting system whidi permits a real image obtained by a real-time 
imaging device (e.g. a video camoa) to be merged with a corresponding 
virtual image generated £rom the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration widi one another, and 
whereby ^s registration will be automatically maintained even as the 
real-time imaging device is moved about relative to the anatomical site, with 
the virtual image being automatically generated so as to follow die real image. 

Still another object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be merged with a corresponding 
virtual image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one another, and 
whereby this registration wiU be automatically maintained through the use of 
a computerized position and orimtation tracker coimected to the imaging 
device even as die real-time imaging device is moved about relative to the 
anatorrdcal site, with the virtual image being automatically generated so as to 
follow die real image. 

Still anodier object of die present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
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imagiiig device (e.g. a video camera) to be merged vnnk a coxrespondrng 
virtaal image generated from the 3-D computer model, whereby the two 
images will be simultaneously displayed in registration with one anofh^, and 
whereby fliis registration will be automatically maintained through the use of 



a computer search algorithm based on die real image and the virtual image 
even as die real-time imagmg device is moved about relative to the anatomical 
site, with the virtual image being automatically generated so as to follow the 
real image. 

Yet anotiier object of the present invention is to provide a video-based 
surgical targeting system which permits a real image obtained by a real-time 
imaging device (e.g. a video camera) to be meiged widi a corresponding 



virtual 



two 



images will be simultaneously displayed in registration with one anodier, and 
whereby a suigical instrument can be tradced along with the real-time imaging 
device in order diat the surgical instrument can be guided about the anatomical 
site using the virtual image generated from the 3-D computer model. 

And ano&er object of the present invention is to provide a video-based 



surgical targeting system which permits a virtual image to be generated from 
the 3-D computer model, and further wherein this virtual image can 
correspond to an "ov^ the shoulder" view of the working tip of a surgical 
instrument located at die axuitomical site. 
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And another object of ibs present inventiQii is to provide an inquroved 
metiiod fox locating anatomical slrnctiiFes during a medical procedure. 

1 

Stnmnfliy Of The Iiivention 

Hiese and other objects of the present invention are achieved through the 
provision and use of a novel video-based surgical targeting systnn. 

In one foim of tiie invention, the video-based suigical targeting system 
comprises apattent-spedfic database comprising a plurality of 2-D images of 
the anatomical structure of a patient; a patient-specific 3-D computer model 
of the anatomical structure of the patient the patient-specific 3-D computer 
model being assembled fi'om the plurality of 2-D images contained in tiie 
patient-specific database; marker placement means for (i) inserting virtual 
planning madcers into the 2-D images contained in the patient-specific 
database^ and/or (ii) adjusting the positions of virtual planning maikers 
inserted into the 2-D images contained in the patient-specific database and 
thereafter incoiporated into the patient-specific 3-D computer model, or 
inserting virtual planning markers into die 3-D computer model; an image 
genmtor for generating a virtual image of the anatomical structure modeled 
by fte patirat-specific 3-D concenter model; real-time image generating means 
for generating a real image of the anatomical structure of a patient; video 
mixing means for mixing the virtual image and the real unage into an output 
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mage; iwiieieby the oiitjput image may comprise eitliar one of the two images 
exclusive of the other» or a composite of boft images; and di^lay means for 
displaying the output image. 

Preferably die video-based surgical targeting system also includes 
registration means for placing the virtual image in registration with the real 
image. These registration means may include means for manually aligning the 
virtual image with the real image, or means for automatically aligning the 
virtual image with the real image. The registration means may also include 
tracking means for tracking the position of the real-time image generating 
means. 

Preferably, the video-based surgical targeting system further comprises 
instrument tracking means for tracking tiie position of a surgical instrument 

In an alternative form of tiie invention, the video-based surgical targeting 
system comprises a patient-specific database comprising a plurality of 2-D 
images of the anatomical structure of a patient; marker placement means for 
inserting virtual planning maricers into the 2-D images contained in the patient- 
specific database; an image generator for generating a virtual image of the 
anatomical structure represented by the plurality of 2-D images contained in 
die patient-^pedfic database; real-time image generating means for generating 
a real image of the anatomical structure of a patient; video mixing means for 
mixing the virtual image and the real image into an output image, whereby the 
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output image may comprise either one of the two images exclusive of the 
other, or a composite of both images; and display means for displaying the 

m 

output image. 

In another alternative form of the invention, the video-based surgical 
taigeting system comprises a patient-specific database comprising a plurality 
of images of the anatomical stracture of a patient; a patient-specific 3-D 
computer model of the anatomical stmcture of the patient, the patient-specific 
3-D computer model being assembled firom the plurality of images contained 
in the patient-specific database; marker placement means for (i) inserting 
virtual planning markers into the images contained in the patient-specific 
database, and/or (ii) adjusting flie positions of virtual planning markers 
inserted into the images contained in the patient-specific database and 
thereafter incorporated into the patient-specific 3-D computer model, or 
inserting virtual planning markers into the 3-D computer model; an image 
generator for generating a virtual image of the anatomical structure modeled 
by the patient-specific 3-D computer model; real-time image generating means 
for generating a real image of the anatomical stmcture of a patient; video 
mixing means for mixing the virtual image and the real image into an output 
ymagft^ whercby the output irruige may comprise either one of the two images 
exclusive of the other, or a composite of both images; and display means for 
displaying said output image. 
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Inyet another alternative fonn of die invention^ die video-based surgical 

tugeting system con^nises a patient-specific database comprising a plurality 

* 

of 2-D images of the anatomical structure of a patient marker placement 
means for inserting virtual planning madcers into the 2*D images contained in 
die patient-specific database; an image generator for generating a virtual image 
of die anatomical structure defined by the patient-specific database; real-time 
image generating means for generating a real image of the anatomical structure 
of a patient, the real-time image generating means being adapted so as to 
dynamically update the patient-specific database via the real image; video 
mixing means for nuxing the virtual image and the real image into an output 
image, whereby die output image may comprise eidier one of the two images 
exclusive of the other, or a composite of both images; and display means for 
displaying said output image. 

The present invention also comprises a method for targeting an anatomical 
structure using the video-based surgical targeting system. 

Brief Description Of The Drawings 

These and other objects and features of the presmt invention will be more 
fully disclosed or rendered obvious by the following detailed description of die 
preferred embodiments of the invention, which are to be considered together 
with die accompanying drawings wherein like numbers refer to like parts and 
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fiirthear wiierein: 

Fig. 1 is a schematic view of the major components of a video»based 
suigical targeting system formed in accordance with tiie presrat invration; 

Fig. 2 is a view of an exemplaiy patient-specific 2-D image of the sort 
contained in the patient-specific database; 

Fig. 3 is a view of an exemplaiy patient-specific 2-D image of the sort 
contained in die patient-specific database, wherein the image has had several 
virtual planning markers placed into the imi^e; 

Fig. 4 is a composite view of (i) a real image obtained by a real-time 
imaging device (e.g. a video camera^ and (ii) a virtual image generated from 
the 3-D computer model, wherein the two images are not yet in registration 
with one another. 

Fig. 5 is a composite view of (i) a real image obtained by a real-time 

■ * 

jmqg tn g dcvicc (cg. a vidco camera), and (ii) a virtual image generated from 
the 3-D computer model, wherein the two images have been placed in 
registration with one another; 

Fig. 6 is a view like that of Fig. S, except that the virtual image generated 
from the 3-D computer model has been sliced away to expose tiie virtual 
planning markers; 

Fig. 7 is a view like that of Fig. 5, except that the virtual image generated 
from the 3-D computer model has been faded away to expose the virtual 
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planning maikeis; 

Fig. 8 is a schematic view showing &e positional relationships between 
various elements of the video-based singical targeting systmi and the patient's 
anatomical stmctore, as well as the positional relationship between an 
exen^laiy surgical instrument and die patimtfs anatomical stracture; 

Fig. 9 is a virtual image generated from the 3-D computer model, showing 
a virtual "over the shoulder" view from the tip of a tracked surgical instrument; 
and 

Fig. 10 is a flowchart illustrating one way of operating a video-based 
surgical targeting system formed in accordance widi the present invention. 

Detailed Description Of The PrefeiTed Embodiments 

Looking first at Fig. 1, the starting point for the present invention involves 

■ ■ * ■ 

a data acquisition device 5 which is adapted to generate patient-specific 2-D 
images. For example, data acquisition device 5 might comprise an MRI 
device, a CT scarmer or any odier scanning device of the sort adapted to 
provide a series of 2-D images of the patient* s anatomical structure, where 
each 2-D image corresponds to a plane or slice taken through the scanned 
structure. Alternatively, data acquisition device 5 might comprise an X-ray 
machine or other imaging machine adapted to provide a set of 2-D images of 
the patient's anatomical stmcture, where each 2-D image corresponds to a data 
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set relating to the scanned stnictme; 

The patient-specific 2*D images generated by data acquisition device 5 are 
stored in a patient-specific database 10. Patient-spedfic database 10 is 
constructed so tiiat the patient-*specific 2-D images can be accessed 
individually or in any particular sequmce desired. Preferably patient-specific 
database . 10 comprises an appropriate set of data contained in a computer 
storage system. 

The patient-specific 2-D images stored in patient-specific database 10 are 
used in building a patient-specific 3-D computer model IS. Patient-specific 
3-D computer model IS may be built from the infomiation stored in 
patient-specific database 10 using any one of the many algorithms well known 
in the art, so long as the patient-specific 3-D computer model is capable of 
generating virtual images of the patients anatomical structure from 
substantially any location where a real camera may be positioned relative to 
that anatomical structure during a given procedure. Alternatively, patient- 
specific 3-D computer model IS may be built fi^om the information stored in 
patient-specific database 10 using any one of the many algorithms well known 
in the art, where the patient-specific 3-D computer model is capable of 
generating virtual images of the patient* s anatomical stmcture firom a 
predetermined set of possible virtual camera positions. Preferably, 
patient-^dfic 3-D computer model IS is constructed so that it is capable of 
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generating virtual images of the patient's anatomical structure from any 
location whatsoever, with any desired field of view, and including ai^ other 
camera-specific criteria desired (e.g. particular camera characteristics such as 
focus, optical characteristics, optical distortions, etc.). By way of example, 
one possible algorithm for bofldmg the patient-spedfic 3-D computer modd 
out of the information stored in the patient-specific database 10 might be die 
Marching Cubes algorithm. 

In one prefeired embodiment of the invention, patient-specific 3-D 
computer model IS comprises an appropriate computer software model 
resident on an appropriate digital computer. Patient-specific 3-D computer 
model 15 is preferably constructed using a plurality of polygonal models to 
model the patienfs real anatonucal stmcture. As is well known in die ait, such 
polygonal models generally comprise a collection of points defining the 
surface of the 3-D comouter model alone with some connectivitv 




to how these surface points are connected to one another. Preferably 
tiiere is one polygonal model per anatomical structure being modeled. Of 
course, the choice of which particular computer software model is used to 
model a particular anatoixucal structure ultimately dqpends on a variety^ of 
factors, such as the particular aiuitomical structure being modeled, the 
computer hardware available, the volume of data to be handled, the particular 
medical procedure which is to be conducted, etc. 
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Of couise, it is also anticipated that die patient-specific 3-D computer 
model could be dynamic as well as static. For example, dynamic changes 
could occur in the model as a result of computer simulation (e.g. through tiie 
use of a Finite Element Model) or because more current real-time data (e.g. 
from video camera 45) is used to iq)date the model, or both. 

It is also anticipated that patient-specific 3-D computer model 15 might 
include data from sources oflier than the 2-D images contained in 
patient-specific database 10. For example, patient-specific 3-D computer 
model 15 might include information obtained from a 3-D sur&ce digitizer such 
as that used in certain types of cranial surgery, or patient-specific 3-D 
computer model 15 might include information obtained from a temperature 
probe or a chemical probe. In fact, in addition to the 2-D image data obtained 
from patient-specific database 10, patient-specific 3-D computer model 15 
might include substantially any type of information gathered by ahnost any 
type of device or sensor or process. 

An integral part of the present invention involves the placement of virtual 
planning markers into tihc patient-specific data, using apparatus 20. This may 
be done by placing such virtual planning markers into one or more of the 
patient-specific 2-D images contained in patient-specific database 10 before 
patient-spedfic 3-D computer model 15 is created; or by placing such virtual 
planning markers mto patient-specific 3-D computer model 15 after tiiat model 
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has been created; or by some combinadon of Ihe two. 

Moie paxticulaify, qqpaiatiis 20 preferably comprises computer hardware 
and software adapted to allow a physician to access one or more of the 
patient-specific 2-D images contained in patient-specific database 10 and 
present Ifaem for iMwing, in the manner shown in Fig. 2 Then, using a mouse 
or other data entry device, the physician can place one or more virtual 
planning markers 25 into an accessed patient-specific 2-D image, in die 
manner shown in Fig. 3. These virtual planning markers 25 can consist of 
substantially any geometric fonn such as a point, a line, a circle, a plane, a 

^ 

path (either straigiht or curved), etc., and are positioned about anatomical 
structures of particular interest to the physiciao. For example, virtual planning 
markers 25 might be placed about a suspected tumor, or they might be placed 
about particularly sensitive anatomical stnictures such as vascular tissue or 
nerve bundles, etc. Naturally, the particular geometric form chosen for a 
particular virtual planning marker may depend on a variety of factors, such as 
die particular anatomical sttucture involved, die computer hardware available, 
the volume of data to be handled, die particular medical procedure to be 
conducted, etc. Regardless of the particular form of virtual plarming marko: 
chosen, each virtual planning marker 25 which is placed on the 2-D image 
using apparatus 20 is then stored in patient-^ecific database 10 along with its 
associated 2-D image. As a result, when patient-specific 3-D computer model 
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15 is suliseqiieiifly created fium the data contained in patient-specific database 
10, patient-spedfic 3-D computer model 15 will incorporate virtual planning 
markers 25 into die model 

Altemalively, a physician may use apparatus 20 to access patient-specific 
3-D c(unputer model 15 and then, using a mouse or oflier data entry device, 
adjust the position of one or more of the virtual planning markers 25 which 
were previously placed into patient-specific database 10 (and thus 
incorporated into patient-specific 3-D computer model 15 when that 3-D 

* ■ > 

computer model was created). Hie adjusted positions of these virtual planning 
markers 25 are then stored in patient-specific 3-D computer model 15. 
Furthemioie, an updated 2-D image of these virtual planning markers can then 
be incorporated into the patient-specific database 10. 

Furthermore, where patient-specific 3-D computer model 15 has not 

ft 

already had a particular virtual planning marker 25 incorporated therein, a 
physician may also use apparatus 20 to access patient-specific 3-D computer 
model 15, and then use a mouse or other data entry device to insert one or 
more virtual planning markers 25 directly into patient-specific 3-D computer 
model 15. 

Once virtual planning markers 25 have been properly positioned in 
patient-specific 3-D computer model 15, virtual images incorporating these 
virtual planning maikers can be created by an image generator 30, fed through 
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a video mixing device 35, and Ihen presented to the physician on display 40. 

At the sometime, real images obtained fiom a video canien45 can be fisd 
flnou^ video mixing device 35 and flien presented to tibe physidan on diq>lay 
40. 

More paiticiilaiiy, image generator 30 comprises compater hardware and 
software of die sent well known in Ifae art for gdierating virtual images from 
patient-specific 3-D computer model 15. 

Video mixing device 35 comprises a video mixing device of the sort well 
known in die art, vrfiereby the surgeon can selectively display (i) a virtual 
unage created by image generator 30 from patient-specific 3-D computer 
model 15, CIO a real image obtainied from video camera 45, or (iii) a composite 
of the virtual image and the real image, where Ae virtual image is 
superimposed against the real image. 

In order for the foregoing composite image to be more useful to the 
physician, the virtual image generated from patient-specific 3-D computer 
model 15 should be placed into registration with the real image obtained from 
video camera 45. This is done in several steps. 

First, and looking now at Fig. 4, a virtual image 50 is created by image 
generator 30, fed through video mixing device 35, and presented on display 
40. Simultaneously, a real image 55 is captured by video camera 45, fed 
through video mixmg device 35, and presented on display 40. Video mixing 
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device 35 is arranged so as to present vutnal image 50 and real image 55 
simultaneously on display 40» widi the virtual image beixig supmmposed on 
tiie real image (Le^ so that tlie virtual image is in the foreground and the real 
image is in the background, in the manner shown in Fig. 4). 

Second, and looking now at Fig. 5, the two images are placed into proper 
registration with oat another. This image registration can be accomplished in 
dther one of two ways. 

* 

A first, and generally more preferable, technique involves holding the 
position of video camera 45 (and hence real image 55) constant and moving 
tiie position of the **virtual object" or the "virtual camera" by means of 
apparatus 57 until die virtual image 50 is brought into registration with real 
image 55. This can be done automatically by having image generator 30 use 
a search algoiidun to match the virtual image to ^e real image, in which case 
apparatus 57 includes computer hardware and software of the sort well known 
in die art to cause image generator 30 to work through a search algorithm to 
match the virtual image to the real image; or it can be done manually, in which 
case apparatus 57 includes computer hardware and software of the sort well 
known in die art to allow the physician to drag the virtual image 50 into 
registration with real unage 55, using a mouse or other data entry device 
attached to image generator 30. One convenient way that mouse motion can 
be used to ccmtrol the 3-D movements of the patient-specific model is to map 

\ ■ ■ ' I 
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such motion into vectors defined by the view direction of the virtual camera, 
as will be well known to persons skilled in die art " 

A second, and generally less preferable, technique involves holding the 
virtual image SO constant and moving video camera 45 (or the patient) until 
real image 55 matches virtual image SO. 

Regardless of which technique is used, once virtual image 50 has been 
matched to real image 55, the position of the "virtual camera" (i.e., the location 
from which the virtual image appears to be seen) will be matched to the actual 
position of video camera 45 . 

Once virtual image SO has been placed into proper registration with real 
image 55, image graerator 30, video camera 45 and video mixing device 35 
can be used to present the virtual and real images on di^lay 40 in various 
presentation formats so as to facilitate a particular medical procedure. In 
particular, one can use image generator 30, video camera 45 and video mixing 
device 35 to superimpose a virtual image (generated from patient-specific 3-D 
computer model 15) against a real image (generated by video camera 45), with 
image generator 30 being directed to modify the virtual image so as to expose 
one or more of the virtual planning markers 25 present in patient-specific 3-D 
conqmter model 15, whereby the anatomy highlighted by virtual planning 
markers 25 will be brought quickly to the attention of the physician. 

In this respect it is to be appreciated that since virtual planning markers 
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25 iwill fieqaently be located ivithia interior portions of the structure modeled 
by patimt-^ecific 3-D computer model IS, exposing the otherwise-hidden 
virtual planning markers to view will involve rendering some or all of the 
virtual anatomical simcture transparent or semi-transparent. This can be done 
eitiier by (i) slicing away any portions of the virtual image required in ordor 
to expose virtual planning maikers 25, whereby those virtual planning maikers 
will be rendered visible against the background real image, in the manner 
shown in Fig. 6; or (ii) fading away some or all of the virtual image so as to 
expose such virtual planning markers against the real image, in the manner 
shown in Fig. 7. As a result; a physician can then conduct a medical 
procedure widi the ccxofidence of using real images generated by video camera 
45, while having virtual planning maikers 25 superimposed against the real t 
image of the anatomical structure so as to help guide the procedure. 

"While the foregoing system constitutes a major improvement over tiie 
prior art, it has also been recognized that it can itself be improved upon 
significantly by adding camera tracking means to video camera 45. 

More particularly, in many medical procedures, the position of video 
camera 45 may change on a fairly frequent basis. Since it is important for 
virtual imaj^ 50 (gmerated from patient-specific 3^D computer model 15) to 
remain in proper registration with real image 55 (obtained from video camera 
45), it is necessary for the video-based surgical targeting system to reestablish 
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proper corresspondence between the two images each time the video camera 
moves. As noted above, diis prppo" conespondence can be reestablished each 
tune video camera 45 moves, fay eitfier (i) having the pfajrsician manually drag 
virtual image 50 into registration witfi real image 55, using a mouse or oflier 
data entxy device attadied to image generator 30, or (ii) having the system use 
a search algoridun to match the virtual image to the real image. Whilebodiof 
these procedures are generally capable (assuming that the anatomical structure 
remains stationaiy) of accurately yielding the desired image correspondence, 
each one suffers from a significant disadvantage. In the first case, requiring 
the physician to physically drag &e virtual image into registration with the real 
image each time the video camera moves can be inconvenient where the video 
camera is moved about frequently. In ttie second case, it can take the system 
a substantial amount of time to solve for the correct image correspondence 
when using a search algorithm, even where high speed hardware is utilized. 
Even where speed is not a problem, automated search strategies of the type 
generally used are prone to errors du^ough false matches. 

Accordingly, it has been recognized that by putting a tracker on video 
camera 45, die position of the real camera can be continuously monitored, 
whereby each time the real camera moves, the position of the virtual camera 
can be correspondingly adjusted by image generator 30. In Has way, the 
virtual image gmerated by image generator 30 can quickly and easily be 
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maintained in registration with the real image, regardless of how often video 
camera 45 is moved. 

More particularly, and looking now at Fig. 8, there is shown a video 
camera 45, an anatomical stnicture 60, and a tracker system 65. Tracker 
system 65 comprises a tracker 70 which is attached to video camera 45, and 
a tracker base 75 which defines &e coordinate system of the tracker system. 

In tiiis setting, Mpc can be considered to represent the matrix 
transformation from the patient's anatomical structure 60 to camera 45; 
can be considered to represent the matrix transformation firom camera 45 to 
tracker base 75; and Mpr can be considered to represent the inatrix 
transformation from anatomical structure 60 to tracker base 75. 

is known from the tracker system. Furthermore, once the virtual 
ixnage genoated by image generator 30 has been placed in re^tration with the 
real image generated by camera 45, the virtual camera position will be known 

relative to the virtual anatomical structure, and hence the real camera position 

I 

will be known relative to the real anatomical structure. Thus, real matrix Mpc 
will also be knowrt hi addition, since and Mpc are then both known, it is 
possible to solve for Mpf. Accordingly, the position of anatomical structure 
60 will then also be known within the relative coordinate system defined by 

tiie tracker system. 

In view of the foregoing, the virtual image generated by image generator 
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30 can be qaiddy and easily maintained in registration with the real image, 
regardless of how often camera 45 is moved. 

Fuitlieimoie, anolher tracker 80 can be positioned on a surgical instnunent 
85. In this case, tracker 80 will provide M^, where represents the matrix 
transfomiation fiom tracker base 75 to instrument 85. Accordingly, once 
and Mn are knoiwn, it is possible to solve for Mp,, where represents die 
matrix transfomiation from the patients anatomical structure 60 to instrument 
85. With Mpi known, it is then possible to track the position of surgical 
instrument 85 relative to anatomical structure 60. This is a very powoiul tool, 
since it allows a virtual representation of surgical instrument 85 to be 
accurately added to the properly registered virtual and real unages presented 
on display 40, i.e.» it allows a virtual representation of instrument 85 to be 
shown in proper registration with a real image of anatomical stracture 60 on 
display 40. 

Furdiermore, as the position of surgical instrument 85 is tracked relative 
to anatomical stmcture 60, it is also possible to provide a virtual image of 
surgical instrument 85 as that surgical instrument 85 moves through the 
anatomical structure, even when some or all of that instrument might be 
hiddw from tiie view of video camera 45. For example, and looking now at 
Fig. 9, it is possible to generate a virtual "over die shoulder" view of the distal 
tip of surgical instrument 85 moving through anatomical structure 60, where 
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that tip will be shown in proper eoirespondence to various structures^ e.g. a 
tumor 95, vascular structures 100, etc. It is also possible to generate an 
additional virtual object 90 showing a line extended along the axis of Ate 
surgical instrument 85 so &at the physician can see where the surgical 
instrument would go if moved further along its cummt trajectory. Of course, 
since flie virtual image generated by the video-based surgical targeting system 
will then differ significantly from die real image generated by video camera 
45, video mixing device 35 should be directed to totally supinress the real 
image generated by video camera 45 so that it will no longer be shown on 
display 40, in order to avoid confusing the physician. 

Mntlification s Of The Prefentsd Embodiments 

It is, of course, possible to modify tiie prrferred embodiments disclosed 

... 

above without departing from the scope of the present invention. 

Thus, for example, while only one camera 45 has been disclosed above, 
it should be appreciated that two or more cameras 45 may be used. 
Furthemiore, it should also be appreciated that camera 45 may comprise a 
video camera, or it may comprise an endoscope, or it may comprise some 
other type of real-time image capturing means, e.g. it may comprise an 

ultrasound device- 

Additionally, it should be recognized that the system might be adapted so 
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tfaatpatifint-qiecific database 10 comprises non-2-D images, e.g. database 10 
might comprise one or more images generated by a Cyberware 3-D scanner. 

Alternatively, patient-specific 3-D computer model 15 might be omitted 
fitmi the ^paratus entirely. In tihis case, die virtual images could be gencxated 
from patient-specific database 10 by image generator 30 through the use of a 
volume rendering procedure of die sort well known in the art. 

■ 

It is also anticipated diat one could use a tracked surgical instrument 85 
to determine the location of anatomical structure 60. This can be 
accomplished by using the tracked surgical instrument 85 to engage known 
fiducial points on the anatomical stmcture. Alternatively, a tracked surgical 
instrument 85 can be used to sample multiple surface points located on the 
anatomical structure and tiien use a data matching procedwe to correlate the 
sampled points widi eidier patient-specific database 10 or patient-specific 3-D 
computer model 15. By way of example, a least squares fit might be used to 
correlate the sampled points with the 3-D computer model. 












be used to target objects in non-medical applications, e.g. it coidd be used to 
target objects ccmcealed within the interior of complex madiines, or objects 
(e.g. a conduit) concealed beneath surfiice of a structure (e.g. the floor of a 
building). 

Furthermore, it is possible to place tracking means directly on die patient 
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SO as to track the position of the patient's anatomical stracture. 
Still p&er changes vnH be obvious to a person skilled in the art 

Advantages Of The Present Invention 

Numerous advantages are obtained tiburough the use of tiie present 
invention. 

For one thing, a surgical targeting system is provided to &cilitate 
locating a particular anatomical structure during a medical procedure. 

For another thing, a video-based surgical targeting system is provided 
to facilitate locating a particular anatomical stmcture during a medical 

, t ^ 

procedure. ^ 

And a video-based surgical targeting systmi is provided which 
permits a series of patient-specific 2-D images (obtained by scanning 
patient anatomy using one or more scanning devices of the type described 
above) to be assembled into a 3-D computer model of the patient's scanned 
structure. 

Also, a video-based surgical targeting syston is provided which allows 
a physician to view the aforementioned patient-specific 2-D images on a 
display in any desired access sequence. 

Furthermore, a video-based surgical targeting system is provided 
which allows a physician to assemble a series of patient-specific 2-D 
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ioiages into a patient-specific database, and dien to generate virtual images 
from tiie aforementioned padent-spedfic database, as seen from any desired 
virtual camera position, for viewing on a display. 

And a video-based surgical targeting system is provided which allows 
a plqrsician to generate virtual images from the aforementioned 3-D 
computer model, as seen from any desired virtual camera position, for 
viewing on a display. 

In addition, a video-based surgical targeting system is provided which 
permits a physician to place virtual planning markers about any sites of 
interest while viewing one or more of the aforementioned patient-specific 

2- D images, with diose virtual plamiing markers then being incorporated 
into the 3-D computer model, whereby those virtual planning markers can 
be displayed in their appropriate 3-D positions when^enerating virtual 
images of the 3-D coniputer model. 

And a video-based surgical targeting system is provided which permits 
a physician to place virtual planning markers about any sites of interest 
while viewing virtual images of the 3-D computer model, widi those virtual 
plaiming markers then being incorporated into the 3-D computer model, 
whereby fliose virtual planning markns can be displayed in dieir 
appropriate positions when subsequently generating virtual images of the 

3- D computer model. 
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And a video-based surgical targeting s/stem is provided which permits 
a physician to place virtaal planning markers about any sites of interest 

m 

while viewing virtaal images of die 3-D computer model» widi those virtual 
planning markers then being incoiporated into the 3-D computer model and 
into die database of 2-D images, whereby diose virtual planning markers 
can be displayed in their appropriate positions when subsequently 
gena:atmg virtual images of die 3-D computer model or when subsequendy 
displfQdng 2-D images from die padent-specific database. 

For another dung, a video-based surgical targeting system is provided 
which permits a real image obtained by a real-tinie imaging device (e.g. a 
video camera) to be displayed to a physician, and which permits a virtual 
image generated from the 3-P computer model to be displayed to a 
physician, according to die directive of die physician. 

And a video-based surgical targeting system is provided which permits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be appropriately merged widi a corresponding virtual image generated from 
the 3-D computer model 

Also, a video-based surgical targeting system is provided which 
permits a real image obtained by a real-^time imaging device (e.g. a video 
camera) to be i^propriately merged with a corresponding virtual image 
generated from the 3-D computer models whereby the two images will be in 
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registradoni with one another. 

Furtheimore, a video*based surgical targeting system is provided 



video camera) to be merged with a corresponding virtual image generated 
from the 3-D computer modd, "hereby the two images will be in 



display either one of the two images exclusive of the other, or a composite 
of both images simultaneously. 

And a video-based surgical targeting system is provided which pemiits 
a real image obtained by a real*time imaging device (e.g. a video camera) to 
be merged with a coiTeq)onding virtual image generated from the 3-D 
computer model^ whereby Iht two images will be simultaneously displayed 
in registration widi one anodier, and whereby the physician can modify the 
virtual inuige generated from the 3-D computer model as needed, by 
clipping or &ding, so as to expose the virtual planning markers to view, 
with die virtual planning markers being superimposed on the real image 
goaerated by the real-time imaging device. 

And a video-based surgical targeting system is provided which permits 
a real image generated by a real-tune imaging device (e.g. a video camera) 
to be merged with a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 




registration widi one another, and whereby the physician can choose to 
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in legistratiQa with one another^ and whereby the physician can modify the 
virtnal image graerated from the 3-D computer model so a$ to expost only 
die virtual planning markers to view, widi the virtual planning markers 
being 5q)erimposed on the real hnage generated by die real-time imaging 
device. 

Moreover, a video-based surgical targeting system is provided which 
permits a real image obtained by a real-time imaging device (e.g. a video 
camera) to be merged with a corresponding virtual image generated from 
the 3-D computer model, whereby the two images will be simultaneously 
displayed in registration with one another, and whereby this registration 
will be automatically maintained even as the real-time imaging device is 
moved about relative to the anatomical site, with the virtual image being 
automatically generated so as to follow the real image. 

And a video-based surgical targetmg system is provided which pennits 
a real image obtained by a real-time imaging device (e.g. a video camera) to 
be merged with a corresponding virtual image generated from the 3-D 
computer model, whereby die two images will be simultaneously displayed 
in registration with one another, and whereby this registration will be 
automatically maintained through the use of a computerized position and 
orientation tracker connected to the imaging device even as die real-time 
imaging device is moved about relative to the anatomical site, with the 
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virtual image being automaticaliy generated so as to follow the leal image. 

Also, a video-abased siugical targeting system is provided which 
permits a real image obtained by a real-time imaging device (e.g. a video 
camera) to be merged with a corresponding virtual image generated fixim 
&e 3-D computer model, whereby die two images will be simultaneously 
displqred in registration vniQi one another, and whereby diis registration 
will be automatically maintained through the use of a computer search 
algorithm based on the real image and the virtual image even as the 
real-time imaging device is moved about relative to the anatomical site, 
with the virtual image being automatically generated so as to follow the real 
image. 

And a video-based surgical targeting system is provided which permits 
a real image obtained. by a real-time imaging device (e.g. a video camera) to . 
be mexged with a corresponding virtual image generated from the 3-D 
computer model, whereby the two images will be simultaneously displayed 
in registration with one anothor, and wherel^ a surgical instrument can be 
tracked along with the real-time imaging device in order that the surgical 
instnunent can be guided about the anatomical site using the virtual image 
gnerated from the 3-D computer model. 

In addition, a video-based surgical targeting system is provided which 
permits a virtual image to be generated from the 3-D computer model, and 
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fiirther wherein this virtual image can coiiespcmd to an "^over the shoulder" 
view of die working tip of a surgical instrument located at the anatomical 
site. 

And an improved mediod for locating anatomical structures during a 
medical procedure is provided. 
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What Is aaimed Tgr 



1. A video-based siugical targeting system comprising: 
a patient-specific database comprising a plurality of 2<-D images of the 
anatomical structmre of a patient 

a patient-specific 3-D computer model of said anatomical structure of 

m 

saidpadenl; said patient-specific 3-D computer model being assembled &om 
said plurality of 2-D images contained in said patient-specific database; 

marker placement means for (i) inserting virtual planning markers into 
said 2-D images contained in said patient-specific database, and/or (ii) 
adjusting flie positions of virtual planning markers inserted into said 2-D 
images contained in saidpaliait-specific database and fliereafter incoiporated 
into said patient-specific 3-D computer model, or inserting virtual planning 
markers into said 3-D computer model; 

an image generator for generating a virtual image of the anatomical 
structure modeled by said patient-specific 3-D computer model; 

real-time image generating means for generating a real image of the 
anatomical stnicture of a patient; 

video mixing means for mixing said virtual image and said real image 
iato an ou^ut image, whereby said output image may comprise either one of 
the two images exclusive of the other, or a composite of both images; and 
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display means for displaying said output image. 

2. A video-based smgical targeting system according to claim I wherein 
said system fistfaer conqnises r^istration means for placing said virtual image 
in registration with said real image. 

3. A video-based surgical targeting system according to claim 2 wherein 
said registration means include means for manually aligning said virtual image 
with said real image. 

4. A video-based surgical targeting system according to claim 2 wherein 
said registration means include means for automatically aligning said virtual 
image with said real image. 

5. A video-based surgical targeting system according to claim 2 wherein 
said registration means include tracking means for tracking the position of said 
real-time image generating means. 

6. A video-based surgical targeting system according to claim 2 wherein 
said system further comprises instrument tracking means for tracking the 
position of a surgical instrument. 
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7. A video-based smgicai taigeting system accoiding to claim 2 wherein 
said system fiulher composes patient tracking means for tracking the position 
of said anatomical stmcture. 

8. A video-based sm:gical taigeting ^stem according to claim 1 wherein 
said real*time image generating means comprise a video camera. 

9. A video-based suigicai taigeting systemi according to claim 1 wherein 
said real-time image generating means comprise an endoscope. 

4 

10. A video-based surgical targeting system according to claim 1 
wheiein said real-time image generating means comprise an ultrasound device. 

11» A video-based surgical targeting system according to claim 1 
wherein said patient-specific 3-D computer model is constructed out of one or 
more polygonal models. 

12. A video-based surgical targeting ^stem according to claim 1 
wherein said image generator generates said virtual image by a polygonal 
rendering process. 
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13. A video-based sorgicai targeting system com 

a patient-specific database comprising a plurality of 2-D images of the 
anatomical structure of a patient 

maiker placement means for inserting virtual planning markers into said 
2*D images contained in said patient-specific database; 

an image generator for generating a virtual image of the anatomical 
structure represented by said plurality of 2-D images contained in said patient- 
specific database; 

real-time image generating means for generating a real image of the 
anatomical stmcture of a patient; 

video mixing means for mixing said virtual image and said real image 
into an output image, whereby said output image may comprise either one of 
the two images excltisive of the other, or a composite of both images; and 

display means for displaying said output image. 

14. A video-based surgical targeting system according to claim 13 
wherein said image generator generates said virtual image by a volume 
rendering process. 

15. A video-based surgical targeting system comprising: 

a patient-specific database comprising a plurality of images of the 
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anatranical stractore of a patient 

a patient-specific 3-D computer modd of said anatomical structure of 
saidpaticnl; said patient-specific 3-D computer model being assembled fiom 

ined in said patient-specific database; 
s for 0 inserting \irtiial planning m 



said plurality of images o 
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the positions of virtual planning maricers inserted into said images contained 
in said patient-specific database and thereafter incorporated into said patient- 
specific 3-D computer model, or inserting virtual planning markers into said 
3-D computer model; 

an image generator for generating a virtual image, of the anatomical 
structure modeled by said patient-specific 3-D computer model; 

real-time image generating means for generating a real image of the 
anatomical structure of a patient 

video mixing means for mixing said virtual image and said real image 
into an ou^ut image, whereby said output image may comprise either one of 
the two images exclusive of the o&er, or a composite of both images; and 

di^iay means for displ^dng said output image. 



16. A video-based surgical targeting system comprising: 

a patient-specific database comprising a plurality of 2-D images of the 
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anatomical stnictore of a patient; 

nuater placement means for inserting virtoal planning maikers into said 
2*D images contained in said patient-specific database; 

an image generator for generating a virtual image of the anatomical 
structure defined by said patient-iqpecific database; 

real-tune image generating means for generating a real image of the 
anatomical stnictare of a patient; said real-time image gmerating means being 

■ 

ad^ted so as to dynamically update said patient-specific database with said 

♦ 

real image; 

video mixing means for mixing said virtual image and said real image 
into an ou^ut image, whereby said output image may comprise either one of 
the two images exclusive of the other, or a composite of both images; and 

display means for displaying said output image. 

17. A video-based surgical targeting system according to claim 2 
wherein said registration means comprise a surgical instrument and tracking 
means for tradking the position of said surgical instrument, and further wherein 
said registration means comprise means for effecting registration when said 
surgical instrument engages known fiducial marks located on said anatomical 
structure. 
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18. A video4)asedsuigicaItraddiig system accoidi^ 
said i^;istntioii means conpise a sui;gical instrument and tracking means for 
tracking the position of said smgical instrument, and fiutiier wherein said 
registxatioa means coiiq>rise means for effecting registration when said 
smgical instnmient sanqiles multiple sui&ce pomts located on said anabmiical 

< 

Structure, with said means for effecting registration using a data matching 
procedure to correlate said sampled points with said patioit-specific database. 



19. A video-based suigical tracking system according to claim 4 wherein 
said registration means are adapted to use said real image to track and update 
said patient-specific 3-D conqyuter model. 



20. A method for targeting an anatomical strocture, said method 

• ■ 

comprising the steps of: 

(1) providing a video-based surgical targeting system comprising: 
a patient-i^ecific database comprising a plurality of 2-D images of 
the anatomical structure of a patient; 

a patient-specific 3-D computer model of said anatomical structure 
of said patient said patient-spedfic 3-D computer model being assembled 
from said phnalily of 2-D images contained in said patient-specific database; 

marker placement means for (i) inserting virtual planning markers 
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into said 2-D images contained in said patient-specific database, and/or (ii) 
adjusting tiie positions of -virtual planning nuukeis inserted into said 2-D 
iniiipiMB contained in said patient-specific database and thereafter incorporated 
into said patient-specific 3-D computn- model, or insetting virtual planning 
markers into said 3-D computer model; 

an im^ generator for generatiiig a virtual image of the anatomical 
structure modeled by said patient-spedfic 3-D computer model; 

real-time image genoratiitg means for generating a real image of the 

anatomical str u ct u r e of a patient; 

video mixing means for nuxing said virtual image and said real iinage 

into an output image, whereby said output image may comprise either one of 
the two images exclusive of die other, or a con^osite of both images; and 
display means for displaying said output image; 

(2) mscrtmg a visual pbnnmg marker into a 2-D image contained in said 

patient-specific database; 

(3) generating a virtual image of the anatomical structure defined by said 
patient-specific 3-D model, and generating a real image of the anatomical 

structure of a patient; 

(4) mixing said virtual iinage and said real image into an output unage, 
whereby said output image comprises a composite of both images, and 
displaying said output image on said display means, wherein said virtual image 
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and said leal image ace xnixed SO tiiat said visual plaim^ 
to view so as to be superimposed against said leal image. 

« 

21. A meifaod according to claim 20 wherein sdd vii^^ 
real image are mixed so as to be in registration with one another. 

22. A method according to claim 21 wherein said video-based smrgical 
taigetmg system conqinses means for maintaining said virtual image and said 
real image in registration with one anodier even when said real-time image 
generating means move relative to said anatomical structm^. 

23. A method for targeting an anatomical structure, said mediod 
comprising the steps of r 

(1) providing a video-based surgical targeting system comprising: 
a patient-specific database comprising a plurality of 2-D images of 
the anatomical structure of a patient; 

maiker placemrat means for inserting virtual plaiming markers into 
said 2-D images contained in said patient-specific database; 

an itnage generator for generating a virtual image of the anatomical 
structure rqnesented by said plurality of 2-0 images contained in said patient- 
specific database; 
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xed-time imagp generating means for genm 
anatomicai stnictnxe of a patient; 

video mixing means for mixing said virtual image and said real image 
into an on^mt image, whereby said ontput image may compii^^ 
the two images excltxsive of &e other, or a composite of both images; and 

display means for displaying said output image; 

(2) inserting a visual plaiming marker into a 2-D image contained in said 
patient-specific database; 

(3) generating a virtual image of tiie anatomical structure represented by 
said plurality of 2-D images contained in said patient-specific database, and 
generating a real image of the anatomical structure of a patient; and 

(4) mixing said virtual image and said real image into an output image, 
whereby said output image comprises a composite of both images, and 
displaying said output image on said displ^ means, wherein said virtual image 
and said real image are mixed so &at said visual plamiing marker is exposed 
to view so as to be superimposed against said real image. 

24. A method for targeting an anatomical structure, said method 
comprising the steps of: 

(1) providing a video-based surgical targeting system comprising: 
a patient-specific database comprising a plurality of images of the 
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anatomical structure of a patient 

a patient-specific 3-D computer model of said anatomical structure 
of said patient, said patient-spedfic 3-D computer model being assembled 
fiom said plurality of images contained in said patient-specific database; 

marker placemoit means for (i) inserting virtual planning maites 
into said images contained in said patient-specific database, and/or (ii) 
adjusting the positions of virtual planning marlcers inserted into said images 
CQUtained in said patient-specific database and thereafter incorporated into said 
patient-^>ecific 3-D computer model, or inserting virtual planning markers into 
said 3-D computer model; 

an image generator for generating a virtual image of the anatomical 
stmcture modeled said patient-specific 3-D computer model; 

real-time image generating means for generating a real image of the 
anatomical structure of a patient; 

video mixing means for mixing said virtual image and said real image 
into an ou^ut image, whereby said output image may comprise either one of 
the two images exclusive of the other, or a composite of both images; and 

display means for displaying said output image; 

(2) inserting a visual plarming marker into an image contained in said 
patient-specific database; 

(3) generating a virtual image of the anatomical structure defined by said 
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patient-specific 3-D model, and generating a real image of the anatomical 
stracture of a patient; and 

(4) mixing said virtual image and said real image into an output image, 
whereby said output image comprises a composite of both images, and 
displqang said ou^nit image on said display means, wherein said virtual image 
and said real image are mixed so that said visual planning maiker is exposed 
to view so as to be superimposed against said real image. 

4 

25. A method for taigeting an anatomical structure, said method 
comprising the steps of: 

(1) providing a video-based surgical targeting system comprising: 
a patient-specific database comprising a plurality of 2-D images of 
the anatomical structure of a patient; 

. ■ 

maricer placement means for inserting virtual planning markers into 
said 2-D images contained in said database; 

an image generator for generating a virtual image of the anatomical 
structure defined by said patient-specific database; 

real-time image generating means for generating a real image of the 
utiytA minal Structure of a padent, said real-time image generating means being 
adapted so as to dynamically update said patient-specific database with said 
real image; 
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video mixii^s ineans for nuxmg said vu^ 
into an oo^t imagp, whereby said ou^ut image may comprise eidier one of 
the two images exclusive of the odicr, or a composite of both images; and 

display means for displaying said output image; 

(2) inserting a visual planniqg marker into a 2-D image contained in said 
patient'-specific database; 

(3) generating a virtual image of the anatomical structure defined by said 
patient-specific 3-D models and generating a real image of the anatomical 
structure of a patioit; and 

(4) mixing said virtual image and said real image into an oulput image, 
in^ereby said ou^ image comprises a composite of both unages, and 
displ^dng said ou^ image on said display means, wherein said virtual image 
and said real image are mixed so that said visual planning marker is exposed 
to view so as to be superimposed against said real image. 
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